Automatic annotation of tennis games: An integration of audio, vision, and learning

a r t i c l e i n f o Fully automatic annotation of tennis game using broadcast video is a task with a great potential but with enormous challenges. In this paper we describe our approach to this task, which integrates computer vision, machine listening, and machine learning. At the low level processing, we improve upon our previously proposed state-of-the-art tennis ball tracking algorithm and employ audio signal processing techniques to detect key events and construct features for classifying the events. At high level analysis, we model event classification as a sequence labelling problem, and investigate four machine learning techniques using simulated event sequences. Finally, we evaluate our proposed approach on three real world tennis games, and discuss the interplay between audio, vision and learning. To the best of our knowledge, our system is the only one that can annotate tennis game at such a detailed level. The rapid growth of sports video databases demands effective and efficient tools for automatic annotation. Owing to advances in computer vision, signal processing, and machine learning, building such tools has become possible [1–3]. Such annotation systems have many potential applications, e.g., content-based video retrieval, enhanced broadcast, summarisation, object-based video encoding, and automatic analysis of player tactics, to name a few. Much of the effort in sports video annotation has been devoted to court games such as tennis and badminton, not only due to their popularity , but also to the fact that court games have well structured rules. A court game usually involves two (or two groups of) players hitting a ball alternately. A point is awarded when the ball fails to travel over a net or lands outside a court area. The task of court game annotation then consists in following the evolution of a game in terms of a sequence of key events, such as serve, ball bouncing on the ground, player hitting the ball, and ball hitting the net. On the other hand, building a fully automatic annotation system for broadcast tennis video is an extremely challenging task. Unlike existing commercial systems such as the Hawk-Eye [4], which uses multiple calibrated high-speed cameras, broadcast video archives recorded with a monocular camera pose great difficulties to the annotation. These difficulties include: video encoding artefacts, frame-dropping due to transmission problems, illumination changes in outdoor games, acoustic mismatch between tournaments, frequent switching between different types of shots, and special effects and …